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ABSTRACT 


This thesis explores the dimensionality of the Defense Language Institute’s (DLI) 
primary student feed back tool — the Automated Student Questionnaire (ASQ). In 
addition a data set from ASQ 2.0 (the newest version) is analyzed for trends in student 
satisfaction across the sub-scales of sex, pay grade, and Defense Language Proficiency 
Test (DLPT) results. 

The method of principal components is used to derive initial factors. Although an 
interpretation of those factors seems plausible, they are subjected to a factor analysis 
rotation (varimax) and five factors are determined and interpreted. Each of the five 
factors are interpreted in terms of student satisfaction with DLI’s: (1) academic 
environment, (2) military environment, (3) non-barracks dormitory living conditions, (4) 
official and supplemental course audio tapes, and (5) service unit’s computer learning 
centers. From the factor loading matrix factor scores equations are developed for use in a 
sub-scale trend analysis. 

Using non-parametric procedures, each factor is checked for differences in central 
tendency by sex, pay grade, and DLPT score (DLPT consists of three tests DLPTL, 
DLPTR, DLPTS). From this analysis the following results derive: (1) sex has no effect on 
any of the factors, (2) pay grade affects satisfaction with the military environment, (3) 
DLPTL, DLPTR, and DLPTS affect satisfaction with the academic environment, and 


DLPTS also affects satisfaction with the computer learning centers. 





TABLE OF CONTENTS 


Ip pe nstre 1B BUCO ey EN Maid ciccsisvassssiseeeeseve i janavensseaePldevssereecy, l 
By, FANCIERS CIID) orate ahr eee ee l 

| DIS Soigstese SV STolit, ine) (Clee: ai hen er eee Ty es | 

2, SHUIGISIT, Sp ane EV OTP aes | 

5), LAVAS 2020 do 1 ene 2 

4. Comoe OL 0G (ls ee 2 

So WESC. coo nae enn es aes ee 3 

©, CH NeR ECO WEIES Scull aite i ee i 4 

JB eo ROTEL L BIL STEN DU 0 4 

Oo MUU OIDY OL CLG een 5 

DL, IJIN SSO NE) ES Gee a GS! IN re 7d 
FIO ess ID) dee fo yh can cvs ea e¢aeacsessdevsesssdsveeeevivensncntbanteataereretaccbevsseonsteee: y 

TOON Yai ANAT CAG | GU coc. chp cvs ennrteedascanetordsasetedebavseesibartensevedeuwe: 7 

2, US IRS OCIS T ISS cake ate ee 8 

BL WEIR LASTED ei OO) se) 3) 8 BO) NS a 10 

Co WAIRTINI CADE SE LOTMA TE7 OY NI a il US Ea ee 10 

L, CORBET COMES US 1c ee eee een eee 13 

2. CGT AXOING TU JESS CLOT 1 ooo 17 

BING eEMiela wom ol metaIMeG COMPONEMUS 2.....0.cccceens scence seeceteseenesssesese scenes: 19 

ID). SEEN OURS UB ONUAN ESCO SS oe 22 
Imlintenpneraeem of Rotated @OmpOnents 2.2. ceccccs ce cctseeeeceedeese sce cess ss. ag 

2h, DWSCUISSTOU, 225.5088 ce re BS 

MGSO OReCmINTRAI MENDENCY ANALYSIS .0.0..00.000. cnc ecccceceteceeeectetieeeeeees oF 
Cee Nee ele eave THODOLOGY soico.ccoooc esr csc eeedestbeteesteeeeees a7 
IBLGSES Me MM PIOVCG 2... 2.502. i Rs, Sc elecnt 27 

PRE OUAlC Cm @olmMpOnelims CONES Of ZETO 40st tes .i..0.,.c2e heen scnekernes cde eee cones: 28 
Peewee eroOr DPET SCORE ON STUDENT SATISFACTION ...................... us 
Seine > Olay GRADE ON STUDENT SATISFACTION.........0..0.0.2.. a 
Pewee bono se ON SPUDENT SATISFACTION LEVELS... 34 

Be ere HON S eC OMIMENDATIONS nooo... ce cates ee eatneeenetennecees 35 
ae ly sie OWinO@NaING ANALY SIS cc. cciccadec ice, coe, heticeswersceseees 35 

JB ALC TONS AUINU ES QS) 0S eI 055: 35 

Ore eee RON ee me 1 Ee NIDBINGIES 2 eek. | Selec encece Peeee eevee 36 
a eer odes ge nas ec sc cseeevewesileanssses sucreecdecceesasdtenedeteeeeeees: ay) 
LES TOQUE 6 ESE) CS SEE | ne rere she 
[INITIAL DISTRIBUTION LIST = 


Seog Oe ot es Goo oo a. Ge” SO Se cee cmc Cue coc ek Name Wi Unt ane be tet paw sie tt Yt ut pC eC Cara Dele Ue Jira eet Duet us Tee Tou ae Dat Tig Dut Tas Jos GL Tee De 2c 2h Tit SL Mi Seo Dds 


Vil 





‘> 


L. INTRODUCTION 


This chapter provides some background on the Defense Language Institute (DLI) 


then addresses the problem this thesis treats and the methodology employed. 


A. BACKGROUND 


This section briefly describes DLI’s mission, vision, goals, student population, 
entrance requirements, language assignments, and proficiency standards. The genesis of 
their primary student feedback tool, the Automated Student Questionnaire, (ASQ) is also 


presented. 
1. DLI — Mission, Vision, and Goals 


DLI’s home page describes their mission in the following way: “To train, sustain, 
and evaluate foreign language skills...” As the world’s “...largest...” and “...most 
respected language resource institute...” their vision for the 21st Century is to remain 
“'. flexible and responsive to the changing world order and on the cutting edge of technical 
innovations...” From their goals statement, those germane to this thesis are as follows: (1) 
Training — 80% of students achieve level two proficiency in listening, reading, and 
speaking, (2) Evaluating — develop valid and reliable assessment tools, (3) Quality 
Philosophy — create a culture of continuous quality improvement, (4) Quality of Life — 
develop and maintain affordable quality of life programs that allow students to focus on 


learning. 
Jee Student Population 


Each year the Defense Language Institute trains thousands of military, DOD, and 
civilian personnel in a wide variety of foreign languages. The Institute plans for 2,900 
students annually and since its inception in 1941 has graduated more than 155,000. This 
student population is screened for a minimum aptitude level prior to enrollment at DLI. 
Military students are first screened for minimum language aptitude via the Armed Services 
Vocational Aptitude Battery (ASVAB). Those that pass the ASVAB’s language aptitude 


section are then evaluated with the Defense Language Aptitude Battery (DLAB). The 
DLAB is based on an artificial language and is designed to measure the student’s ability to 
learn foreign language in general, not his/her ability to learn a specific foreign language. 


3. Language Assignment 


Students are assigned to a specific language based on their DLAB score. Once 
assigned, the specific course of instruction varies in length and difficulty depending on the 
language. Languages are categorized on a difficulty scale ranging from one to four. A 
category one language is a close cognate of the student’s native language (assumed to be 
English) such as Spanish or French. These languages have common words, similar 
sounds, sentence structure, and recognizable symbols (letters). In contrast, category four 
languages are considered “truly” foreign languages and essentially have nothing in 
common with English. Currently 24 languages and dialects are taught and they are 
organized into seven different language schools: Asian 1, Asian 2, Middle Eastern 1, 
Middle Eastern 2, East European 1, East European 2, and West European/Latin 
American. These schools are organized into departments that employ the Team Teaching 
Concept with teams of six instructors responsible for three groups of about ten students 


each. 


4. Completion of Training 


Upon completion of training, students are evaluated for language proficiency by 
another standardized test, the Defense Language Proficiency Test (DLPT). The DLPT is 
best considered as three separate tests: (1) DLPTL, for “listening,” (2) DLPTS, for 
“speaking,” and (3) DLPTR, for “reading.” For each of the three, the scale on which 
proficiency is recorded is an eleven point scale (0, 0+, 1, 1+, 2, 2+, 3, 3+, 4, 4+, 5) on 
which zero represents no proficiency at all and five represents a proficiency level 
associated with a native speaker. As noted in the goals statement, the targeted 
achievement level for the institution is that 80 percent of the students score fwo or better 


on each of DLPTL, DLPTS, and DLPTR. This targeted achievement level can be roughly 


equated to the proficiency level expected from a college language major after two full 


years of study. 
5. The ASQ 


Upon completion of the training process the students are required to provide input 
as to their perception of quality with regard to their overall DLI experience. The data 
collection instrument has traditionally been survey type forms. These surveys evolved 
from paper and pencil to their current fully automated configuration called the ASQ. 
There are two versions of the ASQ, ASQ 1.0 introduced in March of 1993, and the 
current version, ASQ 2.0, implemented in December 1995. Files with ASQ 1.0 data 
contain approximately ten thousand records while files with ASQ 2.0 data contain fewer 
records and number approximately one thousand. Data from both ASQ’s are accessible 
and can be linked to various other respondent items such as: pay grade, class number, 
service component, sex, DLAB, and DLPT score. - 

The ASQ’s purpose is to collect student opinion data on the overall quality of their 
DLI experience. This task is facilitated by dividing the ASQ into two parts, each aimed at 
collecting data on one of two aspects of the students’ entire DLI experience, these aspects 


are as follows: (1) Instructional Effectiveness (IE) and (2) Program Effectiveness (PE). 


a) ASQ1.0 


ASQ 1.0 is further divided into sub-categories within IE and PE. Each of 
these sub-categories attempts to capture student opinion on a particular dimension. These 
dimensions are designed on an intuitive basis and are not disclosed to the respondent 
during the data collection process. Each main category of IE or PE is divided into five to 
eight sub-categories, where each sub-category is supported by one to nine questions. For 
example, one of the sub-categories under IE is “Counseling and Assistance” which is 
supported by three questions which are shown below: 


(1) Went out of his or her way to help me learn the Language. 
(2) Gave me helpful, individual feedback when I needed it. 
(3) Was readily approachable for counseling or assistance when I needed it. 


b)  ASQ20 


The total number of questions and the number of sub-categories in both IE 
and PE are reduced in ASQ 2.0. For the new PE there are now fourteen questions that 
are believed to support only two broad sub-categories: academic and military. The new 


LE arrangement is similarly transformed. 
6. Other Feedback Sources 


DLI staffs internal evaluation teams in the form of Training Assistance Visits 
(TAV) and hosts external committees to aid them in the curriculum review process. 
TAVs and curriculum reviews are expected to provide information leading to better 
understanding of root causes of problems experienced by students. TAV reports are 
qualitative descriptions of the school’s management practices, procedures, and their 
impact on the training program. 

Curriculum reviews were conducted only up to the fall of 1993; however, staffing 
and procedures for re-establishing a revised version of the curriculum review are now 
under discussion. The primary function of the curriculum review was to furnish the 
customer agencies that use DLI graduates a first hand look and direct input into the 


language programs. 


B. PROBLEM STATEMENT 


In accordance with the Institute’s goals of “continuous quality improvement” and 
maintenance of “valid and reliable assessment tools,” DLI staff is evaluating student 
feedback as an indicator of Program and Instructor Effectiveness. They want to know if 
their ASQ really measures the underlying phenomena it purports to measure. Specifically, 


with respect to ASQ 2.0: 


(1) Do ASQ 2.0’s 14 questions support the #wo sub-category design? 


(2) If not, what is ASQ 2.0’s dimensionality and which questions represent those 
dimensions? 


Additionally, DLI’s Research and Analysis Division would also like to see some 
analysis of the ASQ 2.0 responses to determine if student satisfaction levels appear to 


depend on pay grade, sex, or DLPT scores. 


OF METHODOLOGY 


First the ASQ 2.0 data are analyzed to determine the number of separate 
underlying mathematical factors that account for the variability in the data set, in other 
words, to determine the mathematical dimensionality. This is accomplished using the 
method of principal components. Once the principal components are determined, a factor 
analysis is performed in an attempt to improve the interpretations of the components or 
factors. The number of factors are then compared to the number of dimensions believed 
to be designed into the questionnaire. The dimensional design was not disclosed to the 
author until after the factor analysis was completed and interpretations made. The 
mathematical factors are qualitatively analyzed to examine whether they make intuitive 


sense as well. 





i. DIMENSIONALITY ANALYSIS 


This chapter presents an analysis of the ASQ 2.0 PE data. The 14 
questions (variables) are first analyzed for pairwise correlation followed by 
principal components analysis. Several methods for eliminating components are 
discussed; then, using a composite of those methods seven of the fourteen 
principal components are eliminated and five of the remaining seven are 
interpreted. Various numbers of components are retained and subjected to the 
varimax factor analysis rotation scheme and subsequent interpretation of the 
rotated factors is presented. The varimax factor analysis rotation maximizes the 
sum of the squared correlations for retained components thereby producing a 
unique rotation matnx which, when multiplied by the original matnx of 


eigenvectors, produces a unique factor matrix. 


A. THE DATA 


As stated in the introduction, the variables of this data set are the responses 
to questions concerning program effectiveness at DLI, and the respondents are 


DLI students. 
1. The Variables 


There are fourteen variables and the response to each is an integer in the 
range 0-4. The responses represent the students’ opinions on various aspects of 
their entire DLI experience. For each question the possible responses are: 

(1) “0” No opinion or not applicable 

(2) “1” Strongly Disagree 

(3) “2” Disagree 

(4) “3” Agree 

(5) “4” Strongly agree 

All questions are worded so that the higher number response relates to the 


question in a positive way, e.g., “The course objectives were clearly explained to 


me early in the program.” rather than, “The course objectives were not well 
explained.” Therefore, for all questions a higher number is interpreted as better 


than a lower number. The entire set of questions 1s shown in Figure 1. 


. The overall instructional program for my language was well organized. 

. The overall program goals and related requirements were clearly explained. 
. Teaching team cooperation fostered an effective learning environment. 

The grading system was clearly explained early in the program. 

. The official course tapes had good sound quality. 

. The official text materials were useful. 

Course tests appropriately measured my ability. 


. Supplemental (instructor-prepared) language tapes had good sound quality. 


] 
2 
3 
4. 
5 
6 
7. 
8 
9 


. My service unit’s computer learning center was helpful in supporting my study 
program. 

10. My barracks/dormitory was comfortable and well maintained. 

11. My government housing was comfortable and well maintained. 

12 The quality of the food in the dining facilities was good. 

13. My military training at DLI contributed to my overall development. 


14. The care I received at the Troop Medical Clinic was responsive to my needs. 





Figure 1: ASQ 2.0 Questions 


Jay The Respondents 


The set of respondents for this study consists of 615 DLI students. All 
students are unique in that each contributes only one set of responses to the data 
base. The students in the data base are summarized by pay grade and sex in Figure 
2. The vertical axis contains pay grade-sex combinations and the honzontal axis is 


the number of respondents. 
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Figure 2: Number of Respondents by Subsample 


B. VARIABLE CORRELATIONS 


Correlations between pairs of variables are computed from the 615 
observations on each question and are shown below in Table 1. Combinations that 


appear to be “reasonably” correlated are shown in boldface type. 
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Table 1: New ASQ correlation coefficients 


From this correlation matrix several possible groupings of variables are 
readily apparent. Most notable is the group that contains Q1, Q2, Q3, Q4, Q6, 
and Q7. Two other groups that are less well correlated are Q5 with Q8, and the 
group consisting of Q10, Q12, and Q13 


c. PRINCIPAL COMPONENTS 


Principal component analysis is a technique for transforming a set of 
original, interrelated variables into a set of new uncorrelated variables where each 
component is a linear combination of the original variables and the “information” 
contributed by each component is proportional to its variance (Afifi and Clark 


1990). Thus for N orginal variables x 7, X 2... Xy Wwe want to arrive at a set of 


new variables C;, C>,..., Cy where: 


Cj=ayjXp+ 2jX24+...+ AinXy 


subject to the following three constraints: 

(1) The variance of C; = variance C72 ... = variance Cy 
(2) All Care orthogonal 

(3) ForallC ;, @?yjt Qy+..4 @in= 1 


Pnor to performing the principal components analysis the original variables 
are standardized (each is divided by its standard deviation) so that the principal 
components are found for the correlation matrix rather than the covariance matrix. 
According to Afifi and Clark (1990) this simplifies the analysis because: (1) The 
total variance of the system is now equal to the number of variables NV, (2) the 
correlation between the 7” principal component and the j“ variable is simply 
ry=AyV VAR C ;, where VAR C; is the eigenvalue of the *” column. From the 


correlation matrix shown above as Table 1 the principal components of the data 
are calculated and shown below in Table 2. 

In Table 2 each column is an eigenvector and the eigenvalue shown 
represents the variation accounted for by that eigenvector (each eigenvector is a 
principal component). The principal components run sequentially (one through 
fourteen) from left to right in descending order based on the magnitude of their 
eigenvalue. The row labeled “Percent” shows the percent of variation accounted 
for by that particular eigenvector. For example, in column one the eigenvalue, or 
total vanation accounted for by the first principal component (PC-1) 1s 3.09. The 
total variation is equal to the number of variables, in this case 14, making both the 
individual and cumulative percentage of variation accounted for by PC-1 equal to 
3.09/14 which we can see from Table 2 is 22.08 percent (approximately, due to 
rounding). Similarly in Table 2, the row labeled “CumPercent” shows the 
percentage of total variation accounted for by the summation of eigenvalues from 


column one through column fourteen. To calculate the cumulative percentage for 


el 


any column, the eigenvalue of that column and all previous eigenvalues are 
summed and the sum is divided by 14. 

Following Afifi and Clark (1990), variables that have a correlation 
coefficient greater than .5 with a principal component are considered highly 


correlated and said to “load” on that component. For the table of eigenvectors 
shown below this amounts to values of aj; > .5/ 4 VAR C; (where C ; is column i 


from Table 2, and @;; is the entry at the intersection of the i” column and i row); 
these values are shown in boldface type in the body of the table. Again, several 
groupings of component-variable combinations are apparent. Based on Afifi and 
Clark’s criterion we see that variables Q1, Q2, Q3, Q4, Q6, and Q7 load on PC-1. 
This grouping 1s identical to the most prominent grouping surfacing in the analysis 
of the correlation matnx of Table 1. In similar fashion, variables Q10, Q12, and 


Q13 load on PC-2, which also parallels the correlation matrix findings. 
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Table 2: Principal components 


Also of interest 1s that Q14 is represented by PC-7. Q5 and Q8, which 


appeared correlated earlier, do not pass Afifi and Clark’s loading criterion on any 


component, however Q5 correlates with PC-1 at the 7 value of .41, and Q8 with 


PC-3 at an; of .46, both just slightly below the cutoff level. 


1. Component Consistency 


Before performing any further analysis on the principal components, 
verification that the apparent structure really exists must be obtained. If the 
components really exist then they should also be apparent when the data are 
divided into smaller or arbitrary sub-sets. To check the consistency of the 
underlying structure the data are divided using three separate schemes: (1) time, 
(2) number of records, and (3) random sort (each will be described in a separate 
paragraph to follow). To obtain “pictures” of the data structure GH' biplots are 
produced for the data reference set and sub-sets. Biplots project higher 
dimensional spaces onto lower dimensional sub-spaces: in our case the fourteen 
dimensional data space is projected onto a three dimensional sub-space. GH! 
biplots scale the principal components so that the standard deviation of each is 
equal to one. They are better at illustrating the variable correlations than are the 
standard or “JK” biplots. The rays numbered one through fourteen represent the 
original variables and clusters of rays indicate groups of variables that are highly 
correlated. All the GH' biplots shown below give views from the same perspective 


for ease of comparison. 


a) The Reference Picture 


The components and rays derived from the entire data set are used 
to produce the reference picture given by the GH' biplot in Figure 3. The 
variables’ three major groupings are circled and labeled in Figure 3 below. In the 
following comparisons the sub-set biplots are compared to this original picture 
with respect to cluster composition and orientation on the X, Y, and Z axes. In all 
cases the clusters onent as follows: (1) group one points down and to the nght and 


consists of Q1, Q2, Q3, Q4, Q6, and Q7, (2) group two points down and to the 


13 


left and consists of Q5, Q8, and Q11, (3) group three points up in the direction of 
the Y axis and consists of Q9, Q10, Q12, Q13, and Q14. 





Figure 3: GH' Biplot Showing Original Component Groupings 


b) Division by time 


For a first look at an arbitrary division of the data, the median ASQ 
completion date is chosen as the dividing line. This division puts a total of 223 
records in sub-set one, and the remaining 392 records in sub-set two. GH' Biplots 
of the two sub-sets are shown below in Figure 4 with sub-set one as Figure 4a and 
sub-set two as Figure 4b. It is apparent that both divisions of the data have 
roughly the same shape as the reference biplot. In Figure 4a we see the same three 
groupings of rays but with slightly more “distance” among rays within groups as 
compared to Figure 4b; this greater variability may be caused by the smaller 


number of records allocated to sub-set one. 





Figure 4: GH' Biplots of data divided by time 


Cc) Dividing data set in half 


The second arbitrary grouping scheme divides the original, 
unordered data in half (approximately) with 307 records in sub-set one (Figure 5a) 
and 308 records in sub-set two (Figure 5b). Again, GH' biplots are presented 
below and we see the familiar shape of the reference data set reflected in both 
Figure 5a and Figure 5b lending more credence to the existence of a consistent 


underlying structure. 
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Figure 5: GH' Biplots of Data Divided in Half 


d) Random Division 


As a final check for consistency, the “date ordered” data are 
randomly separated into two groups. This separation was effected using the Excel 
RAND function. The division scheme is performed several times with a set of 
biplots created and analyzed for each distinct data division. All divisions evidenced 
similar shapes therefore only one set of biplots is presented below in Figure 6. 
Again, the familiar shape of the reference data set is readily apparent in the 


randomly divided data. 





Figure 6: GH' Biplots of data divided randomly 


e) Conclusions on Consistency 


With three separate data division schemes yielding very similar 
results, the data’s underlying structure is determined to be consistent and further 


analysis is justified. 
2: Component Retention 


One objective of principal component analysis is reduction of 
dimensionality. If reduction is to be realized certain components must be culled. 
Many criteria for reyecting (or keeping) components exist and the ones selected for 
this analysis result from work done by Kaiser(1960), Cattell(1966), Afifi and 
Clark(1 990) Jolliffe(1972), and Dunteman (1989). Kaiser (1960) recommends, for 
correlation matrices, dropping components with eigenvalues less than one. Jolliffe 
(1972) claims, based on simulation studies, that Kaiser’s “less than one” cniteria 1s 
too restrictive and often throws away much information and that for a correlation 
matrix .7 is a better target value for eigenvalues. Cattell (1966) recommends a 


“scree” graph. In the scree graph, eigenvalues are plotted in decreasing order and 


i, 


connected by lines. A point “k” is chosen where the connecting lines are “steep” 
to the left of k and “not steep” to the nght of k; finally k principal components are 
selected. Afifi and Clark (1990) as well as Dunteman (1989) recommend retaining 
enough components to account for a significant cumulative percentage of 
variation; in the examples shown they used 80 percent as a guide. 

Referring to Table 2, the above criteria pose several plausible 
combinations, all yielding similar results. Following Jolliffe (1972), Afifi and Clark 
(1990), and Dunteman (1989) we select nine principal components as the ninth 
eigenvalue is .7006 and the cumulative variation accounted for by nine 
eigenvectors is 80.8034 (under PC-9 in Table 2). Selecting according to Kaiser 
(1960) we choose only four principal components where the fourth eigenvalue is 
1.06, (the fifth is .95) and the cumulative variation accounted for by four 
eigenvectors is 51.8535 (under PC-4 in Table 2). Cattell’s scree technique results 


in the graph shown in Figure 7. 
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Figure 7: Scree Graph of Eigenvalues 


The slope of the scree graph decreases sharply between pmincipal 
components | and 3, and then less sharply between 3 and 6 and appears to 
decrease at a constant rate through the fourteenth component. Based on the scree 


analysis six components seem to be a reasonable choice. The eigenvalue and 
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cumulative percentage of variance for six components are .8318 and 64.59 
respectively (under PC-6 in Table 2). Using the selection methods discussed 
above, a range of four to nine principal components is selected for possible 
retention. Given this range seven components are chosen for the following 


reasons: 

a) Eigenvalue and Cumulative Variation 

The eigenvalue and cumulative variation for the seventh component 
(PC-7 in Table 2) are .8062 and 70.3547 respectively; therefore, its eigenvalue is 


well within the range between Kaiser’s 1.00 and Jolliffe’s 0.7 and together with the 


first six components accounts for over 70 percent of the variation. 

b) Interpretability 

Since the choice of criterion for component retention 1s somewhat 
arbitrary, it seems reasonable to select, from within the aforementioned range of 
four through nine components, the number of components that seems to yield the 


best interpretation. Again that number is seven and the interpretation of those 


seven components follows. 


3. Interpretation of Retained Components 


Seven principal components are retained for the reasons stated above. In 


the context of the original variables the following interpretations are presented: 


a) Component One 


An excerpt from Table 2 (shown below as Table 3) demonstrates 


that variables Q1, Q2, Q3, Q4, Q6, and Q7 are all highly correlated (@;; > 


.5/V¥ VAR C; ) with PC-1. 
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Table 3: Excerpt from Table 2 


From Figure | we see that the questions listed above all have to do 
with the academic portion of the students’ DLI experience. Q1 and Q2 concern 
the overall instructional program’s organization, goals, and related requirements. 
Q4 and Q7 are about the grading system and the fests. Q3 1s about teaching team 
cooperation and Q6 about official text matenals. Therefore PC-1 is interpreted as 


a measure of satisfaction with students’ academic environment. 


b) Component Two 


Again, in Table 4 we see that Q10, Q12, and Q13 are highly 
correlated with PC-2. 
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Table 4: Excerpt from Table 2 


As seen in Figure 1 the questions that load on PC-2 are military 
aspects of a student’s DLI experience. Q10 deals with barracks living conditions, 
Q12 with food in the dining facility, and Q13 with military training while at DLI. 
Therefore PC-2 is interpreted to be a measure of satisfaction with students’ 


military environment. 


c) Component Three 


PC-3 appears to be highly correlated with only one question. 
However, that question lends itself to a unique interpretation in that it relates to 
government housing as opposed to barracks or dormitories. Since only one 
variable loads heavily on PC-3 no excerpt from Table 2 is shown here, however 
the variable’s a,; score is .5804. Since only this question is highly correlated to 
PC-3 and the question concerns a somewhat unique aspect applicable to a smaller 
percentage of students, PC-3 is interpreted to be a measure of satisfaction with 


students’ non-barracks/dormitory government quarters. 
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d) Components Four and Six 


PC-4 and PC-6 have no variables that load on them at the aj; > 


5/V VAR C; level that have not already been used for a component with a higher 
overall variance. For instance, Q11 loads on PC-4 at just about the .5 level; 
however, Q11 has already been allocated to PC-3. PC-4 does have weak negative 
correlations with Q5 and Q8 which appeared to be somewhat correlated in the 
original variables correlation matrix shown in Table 1. This weak negative 
relationship may play a part in the factor analysis phase where it will be seen that 
Q5 and Q8 appear to form their own rotated component with an interpretation 


concerning audio tapes. 


e) Components Five and Seven 


PC-5 has one variable that is highly correlated: Q9. Q9 deals with 
the respondent’s service unit’s computer learning center. Similarly, Q14 singularly 
loads on PC-7. Q14 deals with the care the respondent received at the Troop 
Medical Clinic. Therefore, component five is interpreted as a measure of 
satisfaction with students’ service unit’s computer learning center and seven 1s 


interpreted a measure of satisfaction with healthcare. 


D. FACTOR ANALYSIS 


The purpose of factor analysis is to find new factors that are easier to 
interpret in the context of the original variables (Afifi and Clark 1990). Dunteman 
(1989) points out that if a suitable interpretation can be made from the original 
principal components then there is no need to rotate them. Nevertheless, for 
possible enhancements to our interpretation a factor analysis is performed. For our 
initial factors we restrict attention to the same four through nine principal 
components chosen in the previous section. Various numbers of these principal 
components are subjected to the varimax rotation scheme to See if factor rotation 


contributes anything to our understanding of the data’s underlying factor structure. 
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The varimax rotation maximizes the sum of the squared factor loadings of the 
retained components, producing a unique rotation matrix that when multiplied by 
the original matrix of eigenvectors produces a unique factor matrix. The objective 
is to produce factors that have some loadings near one and some near zero, 
thereby making them easier to interpret. The criteria for a successful factor 
analysis is taken from Johnson and Wiehert (1982) and is known as the “Wow” 
criteria: 
If, while scrutinising the factor analysis, the investigator can 
shout “Wow, I understand these factors!’, the application is deemed 
successful. 


As 1s seen in the following interpretation the “Wow?” criteria is satisfied and the 


rotation adds to the understanding of the data’s structure. 


i: Interpretation of Rotated Components 


The varimax rotation scheme is applied to the candidate range of 
components four through nine. A discussion and interpretation of each is 


presented below followed by a conclusion concerning the rotation process. 


a) Rotating Four Components 


When four components are rotated the rotation matrix groups the 
questions into four distinct groups that differ only slightly from the original 
principal component structure. As with the original principal components analysis, 
Q1, Q2, Q3, Q4, Q6, and Q7 are grouped together in the rotated component one 
(RC-1). The interpretation of this component remains the same as in the principal 
component interpretation: a measure of satisfaction with students’ academic 
environment. 

RC-2 contains all of the questions contained in PC-2 with the 
addition of Q14. This seems to make sense as Q14 pertains to adequacy of care 
received at the Troop Medical Clinic. PC-2 was originally interpreted: a measure 


of satisfaction with a students’ military environment. The Troop Medical Clinic 


23 


is a military activity and therefore its addition to RC-2 serves to strengthen its 
original interpretation. Therefore RC-2’s interpretation is the same as PC-2’s 
interpretation given above. | 

RC-3 also follows the original principal component interpretation: a 
measure of satisfaction with students’ non-barracks/dormitory government 
quarters. Q11, which deals exclusively with government housing, 1s highly 
correlated with this rotated component. 

Unlike PC-4, for which no clear interpretation was discernible, RC- 
4 is highly correlated with Q5 and Q8. Q5 and Q8 both deal with course (and 
supplementary, instructor prepared) tapes’ audio quality: additionally, these are the 
only questions that deal with course tapes. Therefore grouping these two 
questions together seems to add to our understanding of the data’s factor 
structure; RC-4 is interpreted as a measure of students’ satisfaction with the audio 
quality of the official and supplemental course tapes. 

Rotating four components suggests four convenient groupings of 
the variables but it should be mentioned that the four component rotation scheme 
leaves out one variable. Q9, concerning the students’ service unit’s computer 


learning center, 1s not strongly correlated with any of the four rotated components. 


b) Rotating five Components 


When five original components are rotated, the first four rotated 
components are loaded on by the same variables as in the four component scheme 
discussed above, and the magnitude of the loadings do not change appreciably. 
The major change resulting from the additional rotated component 1s that Q9, left 
out in the four component scheme, is now included and turns out to be highly 
correlated with RC-5. Therefore, the interpretation of the first four rotated 
components remains the same as in the four component scheme and RC-5 is 
interpreted to be a measure of students’ satisfaction with their service unit’s 


computer learning center. 
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c) Rotating Six through Nine Components 


The effect of adding a sixth component for rotation is to separate 
two vanables from RC-1. Q6 and Q7 are highly correlated with RC-6. Q6 is as 
follows: “The official text materials were useful.” Q7 reads “Course fests 
appropriately measured my abilities.” The author can see no inherent similarity in 
these two questions. The usefulness of text materials and measurement abilities of 
tests do not seem to be intuitively related. Grouping these two questions together 
— separate from the other questions concerning the overall academic environment — 
seems to add an air of ambiguity to the notion of an underlying structure. Similar 
results occur from the addition of more rotated components. A seventh 
component strips Q14 from RC-2 and isolates the variable as RC-7. This reverses 
the improvement in RC-2 over PC-2. Adding an eighth rotated component 
captures Q4 as that eighth component, further detracting from the integrity of RC- 


1. Finally, a ninth component sets apart the grouping of Q5 and Q8. 
Z Discussion 


The factor analysis indicates that ASQ 2.0 is essentially five dimensional as 
compared to the two dimensions it is designed to cover. Those five dimensions 
are: 

(1) RC-1: a measure of satisfaction with students’ academic 

environment. 

(2) RC-2: a measure of satisfaction with students’ military 

environment. 

(3) RC-3: a measure of satisfaction with students’ non- 

barracks/dormitory government quarters. 

(4) RC-4: a measure of students’ satisfaction with the audio quality of 

the official and supplemental course tapes. 

(5) RC-5: a measure of students’ satisfaction with their service unit’s 


computer learning center 
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For each of the five factors selected, quantitative measures are developed 
for use in further analysis. These quantitative measures are called “factor scores.” 
Afifi and Clark(1990) point out that “it is conceivable to construct factor score 
equations in an infinite number of ways.” The factor scores chosen here are the 
original variable magnitudes scaled by the factor loading coefficient of that variable 
on that factor. This allows the variables with the highest loadings to contribute a 
greater percentage to the total factor score. The factor score equations are given 


below where factor one is labeled “RC-1” and the remainder accordingly: 


(1) RC-1 = .753349Q1 + .724384Q2 + .631647Q3 + 62639104 + 
55760706 + .62927107 

(2) RC-2 = .53546Q10 + .70461Q12 + .70036Q13 + .67486Q14 

(3) RC-3 = .914917Q11 

(4) RC-4 = .71238Q5 + 83231508 

(5) RC-5 = 84917299 
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I. FACTOR CENTRAL TENDENCY ANALYSIS 


This chapter presents analysis investigating the effects of sex, military rank, 
and DLPT score on student satisfaction. DLI’s Research and Analysis Division 1s 
interested in trend analysis across various sub-scales; specifically, questions 
whether student sex, military rank, or DLPT score affect student satisfaction 
levels are considered here. The rotated component scores developed in Chapter II 


are used as quantitative measures of student satisfaction. 


A. NON-PARAMETRIC METHODOLOGY 


Effects of sex, pay grade, and DLPT score on student satisfaction are 
explored using factor scores as response variables. Some of the groupings have 
small numbers of observations associated with them, making accurate 
distributional determinations unlikely; therefore, non-parametric comparisons of 
central tendency are employed, avoiding the normality assumption necessary for 
standard techniques. The data generation mechanism (ASQ’s taken from seven 
different schools with small groups of students taught by various teaching teams of 
six instructors that are not deliberately kept intact from one class to the next) 
provides a defensible guard against serial and auto-correlation; however, the 
random sampling hypothesis is untenable and this analysis suffers to the extent that 


it is violated. 
1. Tests Employed 


The object of this analysis is to determine if a factor’s central tendencies 
are the same across all groups. Mean (and median) scores for all five rotated 
components are tested using three different non-parametric tests: (1) Wilcoxon 
Rank Sum Test (Kruskal-Wallis with more than two groups), (2) Median Test 
(number of points above median), and (3) Van der Waerden Test (normal 
Quantiles). In addition, the data are tested for homoscedasticity using four 
different tests: (1) O’Brien, (2) Brown-Forsythe, (3) Levene, and (4) Bartlett. For 


handling cases of heteroscedasticity the Welch Anova is employed. 
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Je Rotated Component Scores of Zero 


In all cases, rotated component scores of zero are present. The extent to 
which they appear and the effect they have on the analysis vanes by question. 


Table 5 depicts the incidence rates of “0” for all five rotated components. 








ROTATED COMPONENT SCORES OF ZERO 
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Table 5: Incidence of Zero Scores for Rotated Components 


a) RC-1, RC-2 and RC-4 


RC-1, RC-2, and RC-4 all have six percent or fewer zeros, 
indicating that the majority of students answer the questions associated with those 
factors. Nevertheless, to get a zero score for any of RC-1, RC-2 or RC-4 a 
student has to respond “No opinion or not applicable” to at least two questions, 
and in the case of RC-1 that response must be provided for six questions. The 
notion that a student in a military-academic environment either has “no opinion” 
about the school or the military, or that the questions concerning the functioning of 
that environment are “not applicable” to the student seems implausible, and it is 
possible students select zero for some other unknown reason. It is noted here that 
with the high positive response rates for these rotated components the inclusion, or 
exclusion, of the zeros matters little, and since their inclusion is conservative in 


nature they are kept in for this analysis. 


b) RC-3 and RC-5 


RC-3 and RC-5, shown in Table 5, have significant incidence rates 


for zero responses. This is to be expected because RC-3 and RC-5 are both driven 
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by one question each. RC-3 is uniquely associated with Q11, which concerns only 
non-barracks/dormitory government housing, and RC-5 concerns the students’ 
service unit’s computer learning center. Since these questions do not pertain to all 
students the notion of “not applicable” seems plausible here; therefore, the zero 


responses are eliminated. 


B. EFFECTS OF DLPT SCORE ON STUDENT SATISFACTION 


In all three DLPT cases, DLPT score means are found to be significantly 
different with respect to RC-1 (academic satisfaction) and in the case of DLPTS a 
significant difference exists across RC-5. A graphical depiction of central 
tendencies for DLPTL, DLPTR, and DLPTS across RC-1 is shown below as 
Figure 8. 

In Figure 8 quantile boxes are shown to illustrate the response variables’ 
spread and medians; additionally, the jagged lines illustrate the means and are 
connected to highlight the upward trend. The straight horizontal line depicts the 


overall mean. 
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Figure 8: RC-1 vs. DLPTL, DLPTR, and DLPTS 
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Looking at Figure 8 above it appears that there exists an upward trend 
across all three DLPT’s indicating that higher scoring students tend to be more 
satisfied with DLI’s academic environment The test results for RC-1, across the 
three DLPT’s, are presented below in Table 6. The p-values shown in the table 
body indicate the probability of obtaining, by random chance alone, a y? value 
greater than the one calculated, if in fact the distributions across the factor level all 
have the same center. P-values below the .05 level are usually considered ample 
evidence that the distributions do not have the same center. With the exception of 


the Median test for DLPTL, all values are well below the .05 level. In all three 





cases, the DLPT’s heteroscedasticity results indicate that the variances are the 


same across all levels. 
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Table 6: Non-parametric Results for DLPT's vs. RC-1 







In addition to the significant finding for RC-1, it appears that students who 
score higher on DLPTS tend to have a higher opinion of their service unit’s 
computer learning center. Test results, verifying the difference in means for 
DLPTS vs. RC-5, are presented below in Table 7,. There was no indication of 


heteroscedasticity therefore the Welch Anova was not calculated. 
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TEST RC-5 


Median .0046 


Van der Waerden 0018 


Table 7: Test Results for DLPTS vs. RC-5 





Although the test results shown in Table 7 support a difference in means, 
small numbers of observations in the groups making up the “tails” of the sub-scale 


DLPTS, make the notion of a trend somewhat questionable. 


C. EFFECTS OF PAY GRADE ON STUDENT SATISFACTION 


It is noted here that military rank is not the only possible treatment level 
since data from GS employees are also available in the data base. However, RC-2, 
RC-3, and RC-5 all have direct military connotations: RC-2 is interpreted as a 
measure of satisfaction with students’ military environment; RC-3 is a measure 
of satisfaction with students’ non-barracks/dormitory government quarters; RC-5 
is a measure of students’ satisfaction with their service unit’s computer learning 
center. In view of the above RC-2, RC-3 and RC-S5 are considered with only 
military personnel; conversely RC-1 and RC-4 are strictly academically oriented 
and include the GS data 

The non-parametric tests for central tendency with respect to military rank 
indicate that only RC-2 means differ significantly across among ranks. With 
respect to heteroscedasticity, RC-2 is the only factor that indicates significant 
differences in variances across treatment levels and the Welch Anova result of 
<.0001 still confirms the difference in central tendency. As previously stated, 
results for RC-2 are computed without the GS data but with the zero values in the 
data. To further investigate the nature of the dissimilarities among means for 
military rank two more variations are examined and they are as follows: (1) 
military rank without the zero values, (2) military rank without zeros or levels with 


fewer than ten observations. With these modifications, the test results for RC-2 


a2 


continue to indicate that the null hypothesis should still be reyected; therefore, it is 
concluded that significant differences in satisfaction levels exist for various ranks. 
A graphical display of the measures of central tendency for rank versus RC-2 with 


all ranks and zeros included is presented below in Figure 9. 





Figure 9: RC-2 versus Military Rank 


At first glance it appears that there exist two general “levels” for the 
means: (1) enlisted mean at about 5.5 and (2) officer mean at about three. 
However, this is not necessarily the case. It is noted here that pay grades E8 and 
O1 have only one observation each and that O2, 04, O5, and O6 all have fewer 
than ten observations each. When the Tukey Kramer Honest Significant 
Difference (HSD) test is employed we find that there exist similarities from officer 
to enlisted and differences within the enlisted group. The interested reader is 
referred to Appendix A, the matrix of HSD’s where positive values indicate pairs 


of means that are classified as significantly different by that test. 
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D. EFFECTS OF SEX ON STUDENT SATISFACTION LEVELS 


In this case the treatment, sex, has two nominal levels, male and female. It 
appears that neither males nor females evidence a higher level of satisfaction for 
any of the factors — there are no significant results from the non-parametric tests 


for central tendency. 
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IV. CONCLUSIONS/RECOMMENDATIONS 


This chapter provides conclusions concerning the results from the various 
analyses performed in this thesis. Recommendations for possible use of the factor 


scores and suggestions for further study are also provided. 


A. PRINCIPAL COMPONENT ANALYSIS 


Although the principal components found here are interpretable, it is not 
recommended that their interpretations be used to describe the dimensions in the 
data. The principal components are useful for selecting initial factors for rotation; 
however, in this case, the rotated components provide a more thorough and 


intuitively defensible interpretation. 


B. FACTOR ANALYSIS 


The factor analysis performed here is deemed successful, and the author 
has a high degree of confidence in the interpretations provided for the five rotated 
components chosen. It is recommended that the two dimensional ASQ 2.0 
summary report currently used at DLI be modified to include the additional 
dimensions descnbed in Chapter II of this thesis. The variables that load on RC-3, 
RC-4, and RC-S are better correlated with those factors than with the two factors 
from DLI’s original design (academic and military). 

It is recommended that a similar analysis be performed on the ASQ 1.0 
data. Ifthe same factors are determined to exist in that larger data set, the issue of 
trends across the sub-scales (sex, pay grade, and DLPT) can be more thoroughly 
explored, due to the larger number of observations that exist for ASQ 1.0. 
Additionally, this would allow the possible combination of the data bases, or at 
least a portion of them, yielding larger numbers of observations for any future 
studies of this nature. 

It is further recommended that DLI consider modifying its ASQ response 


set to include a “central scale” variable such as “neither agree nor disagree.” This 
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may help eliminate some of the “0” responses that may occur because respondents 
falsely believe that checking “no opinion” is a center scale response akin to the 


suggested “neither agree nor disagree.” 


C. FACTOR CENTRAL TENDENCIES 


The analysis in this section represents a good start toward a thorough 
investigation into trends in student satisfaction based on other population 
characteristics. Obviously sex, pay grade, and DLPT are not the only sub-scales 
worthy of consideration. It is recommended that further analysis be done in this 
area. Other areas of possible interest are the different schools/departments, or 
possibly students’ services. Exploration of a similar nature into these and other 
areas will allow DLI to further their knowledge of their student population, better 


equipping them to succeed in their goal of continuous quality improvement. 
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APPENDIX 
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Oif=Mean{1]-Mean{j] E8 E7 E2 E1 E4 E3 ES E6 O2 6 O3 O1 O4 O5 W1 
=8 000000 023134 024260 051912 122942 1.46972 211618 212875 319344 33261 334689 353406 381710 426630 448487 
E7 -0 23134 = 000000 001126 0.28778 099809 123839 188484 189741 296210 30947 311555 330272 358576 403496 425354 
E2 -0 24260 -001126 000000 027652 098682 122713 187358 188615 295084 30831 310429 329146 357450 402370 424227 
E1 051912 -028778 -027652 0.00000 071030 095061 1.59706 160963 267432 28069 282777 301494 329798 374718 396575 
E4 -122942 -099809 -098682 -071030 000000 024030 0.88676 089933 196402 20968 211747 230464 258768 3.03687 325545 
E3 -146972 =-1 23839 =-122713 + -0.95061 -024030 000000 064646 065903 172372 18568 #187716 206434 234738 =+%279657 301515 
ES -2 11618 -188484 -187358 -159706 -088676 -064646 000000 001257 4.107726 12093 4123071 #£«°+141788 170092 215012 #4236869 
E6 -2 12875 -189741 -188615 -160963 -089933 -065903 -001257 000000 106469 11975 121814 41140531 168835 213754 235612 
a2 -3. 19344 -296210 -295084 -267432 -1.96402 -172372 -107726 -106469 000000 01326 015345 034062 062366 107285 129143 
36 -332611 -309477 -308351 -280699 -209668 -1.85638 -120993 -119735 -013266 00000 002078 020795 049099 094019 115877 
a2 -3.34689 -3:11555 -310429 -2.82777 -2.11747 -187716 -1.23071 -121814 -015345 -00208 000000 018717 4.047021 091941 113798 
oi 353406 -330272 -329146 -301494 -2.30464 -206434 -141788 -140531 -034062 -02075 -018717 Q00000 028304 073224 095081 
04 -381710 -358576 -357450 -3.29798 -258768 -234738 -170092 -168835 -062366 -04909 -047021 -028304 000000 044920 066777 
O5 -4 26630 -403496 -402370 -374718 -303687 -279657 -215012 -213754 -107285 -09409 -.091941 -073224 -044920 000000 021858 
W1 -448487 -425354 -424227 -396575 -325545 -301515 -236869 -235612 -129143 -11587 -113798 -095081 -066777 -021858 000000 
Alpha= 9 95 
Companisons for all pairs using Tukey-Kramer HSO 

q° 
3 40672 
Abs(Dif)-LSO E8 E7 E2 E1 E4 E3 E5 E6 O2 O6 O03 on O4 O5 W1 
E8 -977424 -701743 -6 71802 -644294 -571423 -5 46501 -485357 -4 88358 -453377 -465452 63197 -624018 -351358 -346092 -3 49576 
E7 -701743 -309088 -232523 -205300 -128735 -101983 -047872 -058889 -112675 -1 45489 72528 -394605 030738 -005390 -0 29612 
E2 671802 -232523 -116824 -090028 -007564 022455 065210 044139 -060223 -099142 - 83188 -366916 099509 047062 0 16735 
E1 -6 44294 -205300 -090028 -118530 -036156 -0.06193 036739 015794 -088158 -127040 - 54750 -394712 071468 019128 -011163 
E4 -5.71423 -128735 -007564 -036156 -094491 -063673 -023398 -046132 -155570 -194918 ~94143 -463901 005442 -048284  -0 79042 
E3 -5 46501 -101983 022455 -006193 -063673 -080344 -041768 -065539 -177838 -217417  ~75494 -487040 -016136 -070553 -101541 
E5 -485357 -047872 0.65210 036739 -023398 -041768 -127250 -147557 -2.49367 -288058 -09075 -555187 -090304 -142082 -1 72181 
E6 -488358 -058889 044139 015794 -046132 -065539 -147557 -167627 -258865 -296528 © 31208 -560702 -102752 -151580  -1 80652 
O2 ~453377 = -112675 -060223 -088158 -155570 -177838 -2 49367 -258865 -488712  -5 14603 43523 -7 38659 -360871 -381427 -3 98726 
O6 -465452 =-145489 -099142 -127040 -194918 -217417 -288058 -296528 -514603 -564316 08522 -777268 -418807 -433850  -4 48439 
O03 -363197 072528 183188 154750 094143 075494 -009075 -031208 -343523  -4 08522 36867 -679169 -215802 -266927 -2 96802 
O01 “6 24018 = -394605 -366916 -394712 -463901 -487040 -555187 -560702 -7 38659 -7 77268 79169 -9 77424 = -704764 8-6 99498 = -7 02982 
O04 -351358 030738 099509 071468 005442 -016136 -090304 -102752 -360871 -418807 ~-15802 -704764 -345571 378317 -401129 
O5 -346092 -005390 047062 019128 -048284 -070553 -142082 -151580 -381427 -433850 ~66927 -699498 -378317 -4 88712 -506011 
Ww1 -3.49576 -0.29612 016735 -011163 -079042 -101541 -172181 -180652 -398726 -448439 -96802 -.702982 -401129 -506011 -5 64316 


Positive values show pairs of means that are significantly different 
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